The present invention relates to a monitoring technology for vehicles in which the situation around a vehicle is monitored with a camera for detection of an approaching object.
Conventionally, various approaches have been made for monitoring devices for vehicles for monitoring the situation around a vehicle for detection of an approaching object.
As one of such approaches, there is a technique using an obstacle sensor such as radar. This technique can provide reliable detection of an obstacle, but does not suit to complicate determination such as whether or not the obstacle is coming closer or going away. As other problems, the detection is greatly affected by rainfall and the like, and the detection range is comparatively narrow. For these reasons, it is conventionally difficult to detect an approaching object with an obstacle sensor alone.
A technique using a camera image is also under way. This technique fails to provide reliability as high as that provided by radar, but permits complicate determination such as whether or not an obstacle is coming closer or going away because processing of digitized image information is easy. In addition, a very broad range can be monitored because the detection range is determined by the angle of view and resolution of the camera.
As the technique using a camera image, a stereo method using a plurality of cameras and a method using optical flows are widely known. The stereo method, which uses the parallax between cameras, has problems that calibration between cameras is complicate and that the use of a plurality of cameras costs high.
A monitoring device for vehicles using optical flows is disclosed in Japanese Patent Gazette No. 3011566 (first related art), for example. In this monitoring device, a camera is placed to point to the rear from the vehicle. A screen has a plurality of horizontally-divided regions. In each of these regions, an optical flow is extracted, which has a magnitude equal to or larger than a predetermined threshold and has a direction identical to that of the motion of an approaching object on an image when such an approaching object is assumed. Based on the extracted optical flow, an approaching object is recognized.
For monitoring during curve running, some methods have already been proposed.
For example, in Japanese Laid-Open Patent Publication No. 2000-168442 (second related art), a turning vector is computed from the steering angle and travel speed of a vehicle, and the resultant turning vector is subtracted from an actually determined optical flow, to thereby correct the optical flow. After removal of influence of a curve by this correction, a moving object is extracted.
In Japanese Laid-Open Patent Publication No. 6-282655 (third related art), an optical flow is corrected based on the outputs of a car speed sensor and a yaw rate sensor and pre-measured correspondence between the image position and the distance, to remove influence of a curve. After this removal, a moving object is extracted.
A technique using a focus of expansion (FOE) on an image (see Japanese Laid-Open Patent Publication No. 7-50769, for example) has been widely employed for detection of an approaching object using an optical flow. In some proposals, this technique is applied to curve running.
For example, in Japanese Laid-Open Patent Publication No. 2000-251199 (fourth related art), an optical flow is corrected with an amount corresponding to a travel amount of a FOE.
In Japanese Laid-Open Patent Publication No. 2000-90243 (fifth related art), a screen is divided into a plurality of areas and a virtual FOE is determined for each area based on white-line information determined by a white-line determination means.
Japanese Patent Gazette No. 2882136 (sixth related art) discloses a technique of detecting an approaching object using an optical flow free from influence of a curve. In this technique, a difference ε i2 between a motion vector Vdi theoretically computed from a movement parameter of a camera and a motion vector Vi detected from an image is computed according to the following expression using vectors r1i and r2i representing the reliability of the motion vector detected from an image, to thereby detect a moving object from the computed difference value.εi2=((Vdi−Vi)·r1i)2+((Vdi−Vi)·r2i)2
The conventional techniques described above have the following problems.
The first related art assumes that a vehicle equipped with the monitoring device (user's vehicle) is running straight ahead. Therefore, use of this technique for curve running is difficult. That is, in this technique, an approaching object is detected by use of the “direction of the motion of an approaching object on an image when such an approaching object is assumed”. This “direction of the motion” is not uniquely determined when the vehicle is running along a curve. This matter will be described with reference to FIGS. 34A and 34B.
Assuming that a camera points to the rear, an image as shown in FIG. 34A is taken when the vehicle is running along a curve. In the first related art, horizontally divided region L and region R are set as shown in FIG. 34A. Assuming that an approaching car exists in range AR1 in the region L as shown in FIG. 34B, the “direction of the motion of the assumed approaching car” is lower rightward as shown by the corresponding arrow in FIG. 34B. On the contrary, in range AR2, which also belongs to the range L but is different in vertical position from the range AR1, the “direction of the motion of an assumed approaching vehicle” is lower leftward as shown by the corresponding arrow in FIG. 34B, which is completely different from the direction of the motion in the range AR1. Thus, when the vehicle is running along a curve, the “direction of the motion of an assumed approaching vehicle” varies with the position of the approaching vehicle even in the same region L. This makes difficult the detection of an approaching vehicle.
Even during running of a vehicle along a straight road, the magnitude of an optical flow is significantly different between the upper and lower parts of a screen. More specifically, in the upper part of the screen, which shows a region distant from the user's vehicle, a detected optical flow is very small. On the contrary, in the lower part of the screen, which shows a region very close to the user's vehicle, a detected optical flow is relatively very large.
Therefore, if the upper and lower parts of the screen are subjected to processing with the same threshold, the accuracy of detection of an approaching vehicle is degraded with high possibility. For example, the threshold will be a very small value when determined based on the small flow in the upper part of the screen. If the lower part of the screen is subjected to processing with this threshold, noise tends to arise. On the contrary, the threshold will be a very large value when determined based on the large flow in the lower part of the screen. If the upper part of the screen is subjected to processing with this threshold, most optical flows will be below the threshold, resulting in failure of detection of an approaching object.
In the first related art, also, an approaching vehicle is detected using only optical flows having a magnitude equal to or more than a predetermined value. Therefore, a vehicle running alongside the user's vehicle at nearly an equal speed fails to be detected because the magnitude of the optical flow for this vehicle is nearly zero.
In the second related art, the turning vector generated by turning of the user's vehicle varies in magnitude and direction with the three-dimensional relative position from the camera of an object point related to the turning vector. Therefore, the turning vector can be estimated only when the correspondence between a point on the camera image and a point on the three-dimensional real world is known.
The above matter will be described with reference to FIG. 35. FIG. 35 shows the relationship between a camera image taken with a camera 2 and a three-dimensional coordinate system in the real world. The camera image has Xi axis in the horizontal direction and Yi axis in the vertical direction. The real world coordinate system has Xw, Yw and Zw axes defined as shown in FIG. 35. That is, it is assumed that plane Xw-Zw is a plane parallel to the road surface, direction Xw is rightward and leftward with respect to the user's vehicle, direction Yw is normal to the road surface, and direction Zw is forward and backward with respect to the vehicle. A camera coordinate system (Xc, Yc, Zc) is also set in which the focal point of the camera is the origin and the optical axis of the camera is Zc axis. Note that the directions of the axes are not limited to those specified above. These coordinate systems have relationships given by expression (1) representing perspective projection transformation and expression (2) representing coordinate transformation.
                              Xc          =                      XiZc            f                          ,                  Yc          =                      -                          YiZc              f                                                          (        1        )                                          (                                                    Xw                                                                    Yw                                                                    Zw                                                                    1                                              )                =                              (                                                                                r                    11                                                                                        r                    12                                                                                        r                    13                                                                                        t                    x                                                                                                                    r                    21                                                                                        r                    22                                                                                        r                    23                                                                                        t                    y                                                                                                                    r                    31                                                                                        r                    32                                                                                        r                    33                                                                                        t                    z                                                                                                0                                                  0                                                  0                                                  1                                                      )                    ⁢                      (                                                            Xc                                                                              Yc                                                                              Zc                                                                              1                                                      )                                              (        2        )            where f is the focal distance of the camera and r is constants determined by internal parameters of the camera and by the position at which the camera is placed, that is, the position of the camera coordinate system in the real world coordinate system, which are known values. From the above relational expressions, it is found that a three-dimensional coordinate position in the real world corresponding to an arbitrary point on the camera image is on a straight line passing through the focal point of the camera. However, the position cannot be determined uniquely without further information provided.
To state differently, as shown in FIG. 36, a position in the real world coordinate system can be transformed to a point on the camera image by use of the perspective projection transformation (expression (1)) and the coordinate transformation (expression (2)). However, these relational expressions are not enough to realize the reverse transformation from a point on the camera image to a position in the real world coordinate system. In the second related art, the turning vector must be determined for respective points on the camera coordinate system. However, this transformation is impossible unless other conditions are additionally provided.
In the third related art, as in the second related art, no mention is made of how to determine the correspondence between the image position and the distance, and therefore implementation of this technique as it stands is impossible.
In the fourth related art, a FOE exists only during straight-ahead running, and no FOE exists during curve running in the first place. Therefore, if an optical flow is corrected using a FOE, a fairly large error will arise for running along a curve too sharp to approximate straight-ahead running. This is therefore unpractical.
The fifth related art is naturally unavailable when a vehicle runs on a road having no white line.
The sixth related art, which assumes that immobile objects are predominant on an image, has a problem that detection will be unsuccessful in such a case that a large moving object like a truck exists in the vicinity of the camera.